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Combating the COVID-19 epidemic has emerged as one of the most 
promising healthcare the world's challenges have ever seen. COVID-19 cases 
must be accurately and quickly diagnosed to receive proper medical treatment 
and limit the pandemic. Imaging approaches for chest radiography have been 
proven in order to be more successful in detecting coronavirus than the (RT- 
PCR) approach. Transfer knowledge is more suited to categorize patterns in 
medical pictures since the number of available medical images is limited. This 
paper illustrates a convolutional neural network (CNN) and recurrent neural 
network (RNN) hybrid architecture for the diagnosis of COVID-19 from chest 
X-rays. The deep transfer methods used were VGG19, DenseNet121, 
InceptionV3, and Inception-ResNetV2. RNN was used to classify data after 
extracting complicated characteristics from them using CNN. The VGG19- 
RNN design had the greatest accuracy of all of the networks with 97.8% 
accuracy. Gradient-weighted the class activation mapping (Grad-CAM) 
method was then used to show the decision-making areas of pictures that are 
distinctive to each class. In comparison to other current systems, the system 
produced promising findings, and it may be confirmed as additional samples 
become available in the future. For medical personnel, the examination 
revealed an excellent alternative way of diagnosing COVID-19. 
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1. INTRODUCTION 


COVID-19 has expanded swiftly by person-to-person transfer, wreaking havoc on world health. 
COVID-19 infected around 16.89 million people worldwide, resulting in close to 663,476 deaths [1]. Because 
of the scarcity of intensive care units (ICUs), all countries’ healthcare systems have collapsed. Patients with 
acute illnesses who have been infected with the Coronavirus are referred to ICUs. To prevent the epidemic 
from spreading from person to person, many governments advocated a ‘lockdown’ to ensure population ‘social 
separation’ and ‘isolation’ [2]. The indicators of the coronavirus can range from a cold to a high temperature, 
Shortness of breath, and acute respiratory illness [3]. The most critical step is to detect COVID-19 as soon as 
possible and isolate those who are infected from the rest of the population. The Chinese government has 
identified RT-PCR as a significant indicator for diagnosing COVID-19 patients [4]. It's also, a time-consuming 
procedure with a considerable risk of error [5]. Because of low sensitivity, the coronavirus may not be 
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appropriately recognized in so many cases. Deep learning applications appear to have arrived at the right time 
in the previous five years. Deep learning is a collection of machine learning methods primarily aimed at 
automatically extracting and classifying features from images, with applications ranging from object 
identification to the classification of medical images [6]. Machine learning and deep learning have established 
themselves as established fields for extracting, analyzing, and recognizing patterns from data using artificial 
intelligence. As new data becomes available, reclaiming those fields’ gains for the patient, it's becoming more 
difficult to reap the benefits of clinical decision-making and computer-assisted systems [7]. The role of medical 
imaging technologies in combating this pandemic has sparked a lot of attention [8]. Due to their high 
sensitivity, chest radiographs, including chest X-rays and computed tomography, can both detect COVID-19 
(CT), which is already being examined as a standard detection approach for pneumonia illness [9]. When it 
comes to identifying pneumonia, computed tomography is more reliable than chest X-rays, because chest X- 
rays are less expensive, faster, and use less radiation [10]. To interpret complex medical images, deep learning 
is frequently used in the medical industry [11]. Instead of manually constructed features, deep learning 
approaches rely on automatic feature extraction. It's already conducted a thorough investigation of 
categorization and segmentation issues [12]. In this article, we describe a framework that combines CNN and 
recurrent neural networks for exploiting chest X-rays to identify coronavirus patients. We examined four well- 
known CNN models that had been pre-trained with RNN, including DenseNet121, InceptionV3, and Inception- 
ResNetV2 among the VGG19, in order to identify the optimal CNN-RNN architecture within the dataset 
limitations. The models InceptionV3, DenseNet121, and Inception-ResNetV2. This system's foundation is a 
CNN that was pre-trained to extract important features from images. Then, using the features gathered, we 
applied COVID-19 cases found by an RNN classifier. The article's organization is outlined below. In Section 
2, there is a brief summary of relevant studies. Data gathering, the formation of merged networks, and the 
efficiency evaluation standards are all explained, along with the materials and methods used in Section 3. 
A thorough analysis of the outcomes is provided in Section 4, along with pertinent comments. The article's 
conclusion is presented in Section 5. 

Several attempts were made to build a system for COVID-19 diagnosis utilizing artificial intelligence 
approaches such as machine learning and deep learning in response to the COVID-19 outbreak [11], [13]. This 
section contains the full overview of the newly developed systems for diagnosing COVID-19 patients. Luz et 
al. [14]. To analyze the lung condition utilizing X-ray pictures, and enhanced efficient net convolutional 
network architecture-based model was introduced. A model classified coronaviruses with 93.9 percent 
accuracy and 80% sensitivity using 183 COVID-19 samples. Minaee et al. [15] used 71 COVID-19 samples 
to present a deep transfer learning architecture for distinguishing infected regions from other lung illnesses. To 
distinguish coronavirus cases, the design achieved a sensitivity score of 97.5% as well as a specificity of 90%. 
Khasawneh et al. [16] 284 COVID-19 samples a deep convolutional neural network (CNN) was trained to 
diagnose coronavirus illness. The proposed model detected coronavirus with a precision of 97 percent and an 
accuracy of 89.5 percent. Apostolopoulos and Mpesiana [17] presented a coronavirus infection detection 
technique based on transfer learning. With a precision of 93.48%, a specificity of 92.85 %, and a sensitivity of 
98.75 %, the VGG19 outperformed the competition. Bharati et al. [18]. For the detection of COVID-19, the 
best result for VGG19 was 83 % recall and eighty -three percent precision, which was accomplished using a 
deep transfer learning-based approach. Lakshmanaprabu et al. [19] categorized corona virus-infected patients. 
Researchers employed nine pre-trained CNNs, and a support vector machine was used in a transfer learning 
system. With an accuracy of 95.38%, the ResNet50-SVM outperformed all other models. Karakanis and 
Leontidis [20] COVID-19 detection from the chest X-ray dataset was proposed using the transfer learning 
method. With 89 COVID-19 samples, the system achieved 98.18 percent accuracy and a 98.19% FI score. 
Singh et al. [21] X-ray pictures were used to detect corona virus cases using a CNN and LSTM architecture. 
In this study 421 datasets were used, including 141 COVID-19 cases, then had a 97% accuracy, 91% 
specificity, and 93% sensitivity. 


2. METHODS 

Even thoughmany existing systems have yielded encouraging results, the COVID-19 has a tiny dataset 
because the mutable quality of such datasets stayed not taken into account. It was pointed out that the dataset 
used in person’s studies was exceedingly unrepresentative, which might lead to the majority class being over- 
classified at the price of a minority class being under-classified. Because COVID-19 photos were taken from 
different parts of the world, they were highly inconsistent, while in previous investigations, pneumonia and 
normal photos were homogeneous and well-curated. The COVID-19 samples had the majority of adult patients, 
while the pneumonia dataset contained largely children. Inside the existing systems, these disparities are 
usually disregarded. As a result, our suggested method using the balanced dataset includes photos of both adult 
and children patients to understand a disease's genuine properties. The suggested approach, as depicted in 
Figure 1, includes various phases for COVID-19 infection diagnosis. To begin, X-ray samples from the chest 
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were scaled, in the shuffled and normalized preprocessing pipeline to determine the true features and reduce 
picture noise. The dataset was then divided into exercise and testing groups. 

We selected four CNN architectures with recurrent neural network (RNN) classifiers that have been 
pre-trained on a training dataset. After each epoch, the accuracy and loss of training datasets were computed, 
and the validation loss and accuracy were determined using the five-fold cross-validation technique. The 
confusion matrix was utilized to evaluate the system's performance, accuracy, precision, recall, area under 
curve (AUC), and Fl-score are among the measures available. 


VGG19+RNN, DenseNeti21_ 
+RNN, Inseption-ResNetV2 | 
+ RNN, InseptionV3 + RNN 


sy Precsision 
Recall F1 source AUC © 


Figure 1. The COVID-19 diagnosis framework's overall system architecture [22] 


2.1. Dataset of the study 

In this study, we used a variety of datasets from diverse sources. The COVID-19 Radiography 
database is one of these databases, the CoronaHack-Chest X-Ray dataset, the COVID Chest X-Ray dataset, 
and the COVID-19 X-Ray dataset. The COVID-19 Radiography database is the winner of the COVID-19 
dataset on Kaggle [23]. Qatar University researchers cooperated with clinical professionals to construct a chest 
X-ray picture for COVID-19 in this dataset. These COVID-19 datasets include 1231 normal, 1235 viral 
pneumonia, and 1042 healthy pictures. The COVID Chest X-ray dataset is a public collection of patient chest 
X-rays and CT pictures [23]. A Chinese team created the COVID-19 X-ray dataset to investigate chest CT 
image anomalies [23]. To identify the X-ray pictures, the CoronaHack-Chest X-Ray dataset was created [24]. 


2.2. CNN and deep transfer learning 

In this method, transfer learning the data from one field is moved to a domain with a similar name 
[24]. When a dataset is insufficient to train the variables of any network, this method is used. This section 
provides a brief overview of the CNNs used for automatic recognition. The CNNs that were employed in the 
classification process, as well as the parameters set for transfer learning, are listed in Table 1. The parameters 
were defined after multiple experiments, but there is a plethora of other options that might be studied in a future 
study to see if they contribute to performance improvement. Starting at the bottom of the CNN, the Layer 
Cutoff parameter specifies the number of layers that are not trainable. To extract more information from the 
late convolutional layers, the rest of the layers that are closer to the output features are made trainable. The 
CNN's classifier is located at the top to execute the classification of the features that have been retrieved and 
is referred to by a parameter Neural Network. It is defined by the total number of hidden layers and the total 
number of nodes. 

Some hyper-parameters are shared by all CNNs. The rectified linear unit (ReLU) is responsible for 
activating all convolutional layers [7]. The dropout [25] layer has been added to neural networks with two 
hidden layers to avoid overfitting [23]. The CNNs were generated using the Adam [23] method of optimization. 
Table 1 also illustrates the features of four pre-trained CNN designs, which were trained using a batch size of 
64 for 10 epochs. 

- Inception-ResNetV2: An Inception-ResnetV2 network combines inception and residual connections in a 
single network. Litjens et al. [26], using 164 deep layers. It avoids a degradation problem by using 
different-sized convolution filters that have been trained on millions of photos. 
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- InceptionV3: By extending a network's breadth and depth, InceptionV3 [27] improves computing 
resources. That has 48 layers with missed connections that can be used as a fundamental component. It 
was trained on a million images in 1000 different categories. To reduce dimensionality, Max-pooling is 
used to recreate the inception module. 

- DenseNet121: The dense convolutional network utilizes dense associates between hidden layers rather 
than direct connections [28]. Each layer in the DenseNet architecture is related to the following layer to 
carry data throughout the network. Only a few parameters are used for training the mouth maps, which 
stand sent straight to altogether following layers. For short datasets, thick connections reduce over fitting 
of a model. DenseNet121 is made up of 121 layers, each of which is filled with weights from the Image 
Net dataset. 

- VGGI19: the VGG19 is the deep network architecture-based variant of Karen Simonyan and Andrew 
Zisserman's visual geometry group network (VGG) [29]. To execute on the ImageNet dataset, it has a 
total of 19 layers, 16 convolutional layers and three fully-connected layers are included [30]. VGG19 
employed a three-dimensional with a stride of one, a convolutional filter is used., After that, there is a 
slew of non-linear layers. VGG19 uses max pooling to minimize the image's volume size while 
maintaining good image organization accuracy. 


Table 1. Showsthe characteristics of four CNN architectures that have pre-trained 


Networks Distance _ Parameters (106) 
Inception-ResNetV2 164 56.0 
Inception V3 48 23.6 
DenseNet121 121 8.0 
VGG19 19 144.0 


2.3. Recurrent neural network (RNN) 

The RNN is a sort of extensible feedforward neural network that handles sequential data and has one 
or more feedback loops [31]. An RNN generates an output sequence (y1.,...,yt) using the formulas below, as 
illustrated in Figure 2, given an input sequence (x1.,,...,xt). 


ht=sigm (Whxxt + Whhht-1) (1) 


yt=Wyhht (2) 


Figure 2.Recurrent neural networks' structural [32] 


When a time and capacity-based input-output relationship is discovered, it can be used to manage 
long-term dependency, and RNN is applied [33]. The technique for modeling sequences uses an RNN to 
feedstuff the input arrangement into a course with a defined size, which is then mapped to a softmax layer. 
However, when the gradient vectors grow and shrink inexorably over time, a problem emerges with RNN. 
Because of the vanishing gradient and exploding problems [33], learning long-range associations from RNN 
architecture sequences is difficult. The long short-term memory (LSTM) [34] is a type of short-term memory, 
on the other hand, capable of successfully solvingsuch a long-distance reliance issue. The fundamental 
distinction between LSTM and Recurrent neural networks stays that LSTM has a discrete memory lockup state 
for storing long-term states that it updates or exposes as needed. An LSTM is composed of the following gates: 
inputs gates, output gates, and forgotten gates, at each time step t, ft stands for forgetting gate, Ot for output 
gate, ct for the memory cell, and ht for a hidden state. The LSTM transitional representations are listed in: 
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i=0(wi xt + Uiht-1 + Vict-1) (3) 


f=0(wi xt + Uiht-1 + Vict-1) (4) 
O=0(wi xt + Uiht-1 +Vict-1) (5) 
C=0(wi xt + Uiht-1 +Vict-1) (6) 
Ct=0(wi xt + Uiht-1 +Vict-1) (7) 
h=0(wi xt + Uiht-1 +Vict-1) (8) 


The current input is denoted by xt, osigmoid denotes the sigmoid function, andO element-wise multiplication 
denotes the multiplication of elements. 


2.4. Hybrid CNN-RNN architectures 

In this article, a hybrid technique based on using three different types of X-ray samples, CNN, and 
Recurrent neural networks were created to identify COVID-19. DeneNet121, InceptionV3, and Inception- 
ResNetV3 are VGG19 were used for extract complex features from 224 224 3 sized data. To distinguish 
Normal cases, COVID-19, and pneumonia, the collected features were fed into an RNN classifier. Figure 3 
depicts the COVID-19 classification, CNN-RNN network is used, and includes: 
- Step 1: To extract critical information from X-ray pictures, use a variety of pre-trained CNN models. 
- Step 2: Reorganize the feature map so that it fits in order. 
- Step 3: Set a multi-layered RNN's feature map as the input. 
- Step 4: To categorize COVID-19 chest X-ray pictures, use a softmax classifier. 


Input Images 


Fully-Connected Layer 


Image Pre- 
Processi 
essing Prediction 


COVID-19 Es 
SS ions ii 
Pneumonia, 


Pacumonia 


Figure 3. Shows the COVID-19 diagnosing workflow using the CNN-RNN architecture [35] 


The rectified linear unit (ReLU) was utilized to activate the convolutional layers [36]. The Dropout 
layer [37] was employed to avoid the model overfitting [38]. RMSprop [39] was used to train the CNN 
architectures, which had a total of 150 epochs, a batch size of 32, a learning rate of 0.00001, and a batch size 
of 32. Figure 4 depicts the construction of the hybrid CNN-RNN. 
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Input as X-ray Images 


Feature Extraction 


Figure 4. Illustrations of COVID-19 were diagnosed using a hybrid CNN-RNN architecture 


2.5. Criteria for evaluation 

The developed system's AUC, accuracy, precision, and recall are all utilized to assess its performance. 
The following is a mathematical representation of the evaluation metric parameters. true positive (TP) denotes 
correctly identified COVID-19 instances, true negative (TN) denotes correctly categorized pneumonia or 
normal cases, COVID-19 cases identified erroneously are referred to as false +ve (FP), and false -ve (FN) 
refers to pneumonia or normal instances that have been wrongly classified. 


Recall = TP/ (TP + FN) (9) 
Precision = TP/ (TP + FP) (10) 
Accuracy = (TP + TN)/ (TN + FP + TP + FN) (11) 


3. RESULTS AND DISCUSSION 

The accuracy is shown in Figure 5 during the training and validation periods. At epoch 200, the 
maximal training and validation accuracy of the VGG19-RNN architecture is 97.8 % and 97.74 %, respectively. 
The InceptionV3-RNN network, on the other hand, has the lowest training and validation accuracy (95.14 % 
and 93.01 %, respectively). In this study, a combination of four CNN and RNN stayed used to diagnose 
COVID-19 contamination. The findings showed that VGG19-RNN is more successful than other deep learning 
architectures in identifying COVID-19 patients as pneumonia and normal patients, and it is now regarded as a 
key deep learning architecture. Table 2 shows a comparison of basic CNN-based classifier networks with our 
research. It has been demonstrated that the VGG19-RNN network outperforms CNN networks that have been 
pre-trained. Eventually, Table 3 shows another comparison between current studies and our research. Existing 
technologies can diagnose coronavirus infections with an accuracy of 82.3% to 98.5%, according to research. 
The VGG19-RNN network, on the other hand, achieved 97.8% accuracy, and it improves other current systems. 
The VGG19-RNN networks performed well in comparison to previous research. 


Acuracy 


Number of Epochs 


Figure 5.CNN-RNN architectures’ accuracy 
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Table 2. Using COVID-19 patients, compares the designs of classifier CNN and CNN-RNN 


Classifier Accuracy Precision __ Recall 
DenseNet121 98.04% 98.15% 98.08% 
VGG-19 98.03% 98.01% 98.61 % 
Inception-ResNetV2 98.09% 98.40 % 96.03% 
InceptionV3 98.15% 96.61% 96.82% 
VGG19-RNN 98.39% 98.53% __- 98.08% 


Table 3. Shows a comparison of the proposed CNN-RNN architecture to previous work in terms of accuracy 


Works Design COVID-19Accuracy (%) Accuracy (%) 
Lakshmanaprabu ef al. [19] Xception - ResNet50V2 96.7 90.5 
Islam et al. [22] Xception 94.5 87.3 
Mastouri et al. [23] Tailored CNN 79.0 90.5 
Litjens et al. [26] Inception-ResNetV2 - 90.1 
Yasar and Ceylan [24] DenseNet 77.3 85.6 
Asnaoui [31] Sgdm-SqueezeNet 94.1 96.2 
Chouhan et al. [33] CNN-LSTM - 96.2 
Proposed System VGG19-RNN 97.8 97.8 


4. CONCLUSION 

Using deep learning techniques for COVID-19 diagnosis has become a critical problem during the 
COVID-19 epidemic to deal with a lack of medical resources. The major contribution we used is a combination 
between a CNN and an RNN that uses deep transfer learning to categorize us divided X-ray samples into three 
groups: COVID-19 pneumonia then normal in this paper.The four well-known to extract features CNN 
networks were utilized, and an RNN networks was utilized to categorize distinct classes based on those 
characteristicsalso comparing the suggested model's performance measure to those of existing pertained 
models revealed that the proposed model is more efficient than the others.It should, perhaps, lessen the doctor's 
burden while testing COVID19 patients. Our proposed system does have certain drawbacks. To begin, the 
COVID-19 samples are modest, and additional samples are needed to verify our suggested methodology. 
Second, only a posterior-anterior chest X-ray picture is suitable for this investigation, thus it can't categorize 
other perspectives like apical and so on. Third the results of our test are not likened to those of radiologists 
which is something we want to do in the future. 
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